(12) INTERNATIONAL APPLICATION PUBLISHED UNDER THE PATENT COOPERATION TREATY (PCT) 



(19) World Intellectual Property Organ izati n 
International Bureau 

(43) International Publication Date 
28 June 2001 (28.06.2001) 




PCT 



(10) International Publication Number 

WO 01/47190 Al 



(51) International Patent Classification 7 : 



(21) International Application Number PCT/US00/35154 



H04L 12/26 (74) Agent: GROLZ, Edward, W.; Scully, Scott, Muiphy & 
Presser, 400 Garden City Plaza, Garden City, NY 11530 
(US). 



(22) International Filing Date: 

22 December 2000 (22.12.2000) 



(25) Filing Language: 



(26) Publication Language: 



English 



English 



(81) Designated States (national): AE, AG, AL, AM, AT, AU, 
AZ, BA, BB, BG, BR, BY, BZ, CA, CH, CN, CR, CU. CZ, 
DE, DK, DM, DZ, EE, ES, FI, GB, GD, GE, GH, GM, HR, 
HU, ID, IL, IN, IS, JP, KE, KG, KP, KR, KZ, LC, LK, LR, 
LS, LT, LU, LV, MA, MD, MG, MK, MN, MW, MX, MZ, 
NO, NZ, PL, FT, RO, RU, SD, SE, SG, SI, SK, SL, TJ, TM, 
TR, TT, TZ, UA, UG, UZ, VN, YU, ZA, ZW. 



(30) Priority Data: 

09/469,507 22 December 1999 (22.12.1999) US 

(71) Applicant: MCI WORLDCOM, INC. [US/US]; 515 East 
Amite Street, Jackson, MS 39201 (US). 

(72) Inventors: WELDON, Jed rick, X; 11470 Rothbury 
Square, Fairfax, VA 22030 (US). OSBORNE, Joshua; 
5032 Portsmouth Road, Fairfax, VA 22032 (US). 



(84) Designated States (regional): ARIPO patent (GH, GM, 
KE, LS, MW, MZ, SD, SL, SZ, TZ, UG, ZW), Eurasian 
patent (AM, AZ, BY, KG, KZ, MD, RU, TJ, TM), European 
patent (AT, BE, CH, CY, DE, DK, ES, FI, FR, GB, GR, IE, 
IT, LU, MC, NL, FT, SE, TR), OAK patent (BF, B J, CF, 
CG, CI, CM, GA, GN, GW, ML, MR, NE, SN, TO, TG). 

Published: 

— With international search report. 



[Continued on next page] 



(54) Title: METHOD, COMPUTER PROGRAM PRODUCT, AND APPARATUS FOR COLLECTING SERVICE LEVEL 
AGREEMENT STATISTICS IN A COMMUNICATION NETWORK 



,401 



,406 



,409 



,410 



,407 



PROGRAMMABLE 
PROBE DEVICE 



,477 



Routing 
Engine 



MAW 
MEMORY 



ROM 



STORAGE 
DEVICE 



BUFFER 
UNIT 



BUS 



403' 



\405 



PACKET 
GROUPING 
LOGIC 



,415 



ENVELOPE 
PACKET 
LOGIC 



INPUT/OUTPUT 
UNIT 



417 




IP 

NETWORK 



y450 



Source 



ON 
1—1 

' - (57) Abstract: A probing router (401) is used at a source site of a virtual private network (417). In-band probing operations are 
£Zj performed by components within the probing router, using processing resources available from a router engine portion of the prob- 
ing router. In this way, changes in the network and service level agreement statistic collection processes may be quickly and easily 
accommodated within the probing router. Furthermore, the probing router communicates the probe message through an in-band 
communication channel so as to provide a direct measurement of service level data for the channel used for communication infor- 
^ mation between the source site and a destination site. 



WO 01/47190 Al lIllfillillilllllilllSHINl 



For two-letter codes and other abbreviations, refer to the "Guid- 
ance Notes on Codes and Abbreviations' 1 appearing at the begin- 
ning of each regular issue of the PCT Gazette. 



WO 01/47190 PCT/US00/35154 

METHOD, COMPUTER PROGRAM PRODUCT. AND APPARATUS FOR 
COLLECTING SERVICE LEVEL AGREEMENT STATISTICS IN A 
COMMUNICATION NETWORK 

5 

BACKGROUND OF THE INVENTION 
Field of the Invention 

The present invention relates to apparatuses, methods and computer program 
10 products that collect service level agreement (SLA) statistics in communication 
networks and especially virtual private networks (VPN). 



Discussion of the Background 

Communication networks provide an infrastructure by which messages (digital or 
15 analog) may be routed from a source to one or more destinations. Proprietary, 

exclusive networks may be used when messages are to be distributed only between 
a private set of network nodes. These proprietary networks may span only local 
regions, and are thus called local area network (LAN). Similarly, such proprietary 
networks may extend across a single city, and thus may be referred to as a 
20 metropolitan area network (MAN). When extending over a larger geographic 

region, where the nodes are separated by relatively large distances, the network is 
referred to as a wide area network (WAN). 



However, the expense of establishing and maintaining a proprietary network 
25 whether it be a LAN, MAN or a WAN, is often not cost effective. Furthermore, 
maintaining the network often requires personnel with specialized skills, having 
job descriptions that may be well outside the scope of the company's main line of 
business. While the proprietary network does offer the advantages of dedicated 
security and avoidance of traffic congestion problems, the expense and 
30 maintenance issues associated with developing proprietary, exclusive-use networks 
is not often justifiable, particularly when publicly available resources are available, 
such as the Internet. 

Virtual private networks (VPN) provide a cost effective alternative to proprietary 
networks. A VPN enables communication among a "community of interests" by 
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enabling private traffic to be passed between at least two nodes within the VPN 
using a shared communication resource, such as the Internet. When the Internet is 
used as a component of the communication network, the VPN is referred to as an 
"Internet VPN". However, unlike un-regulated and uncontrolled communications 
5 over the Internet, a VPN is usually established by Internet service providers (ISPs), 
who provide differentiated services from other users who are not part of the VPN. 
The differentiated services for users of the VPN, are contractually governed by an 
agreement between the ISP and VPN customer in th form of a "service level 
agreement" (SLA). 

10 

The SLA may include provisions for a predetermined network availability, such as 
99.9% average end-to-end availability over a one month period for 10 or more 
sites, and at least 99.8% average end-to-end network availability over a period of 
one month for 3 to 9 sites. Network speed is another metric of performance that is 

15 typically part of the SLA, where an average network latency may be specified to be 
120 milliseconds (ms) for round-trip transmission between VPN sites within the 
United States or within Europe, for example. Some Internet service providers, 
such as UUNET will provide a service level guarantee and will credit an account 
of a VPN customer if the level of service, as defined in the SLA, was not achieved. 

20 An optional feature in VPNs is the availability of encryption for data packets so 
that unintended "listeners" will not be able to decipher the information content of 
the messages sent through the commonly available information channel. 

VPNs, and in particular Internet VPNs, often choose to employ tunneling 
25 technology as a way to securely transfer data between two similar networks (e.g., 
private LANs) over an intermediate network such as UUNET net IP network. 
Tunneling (sometimes referred to as "encapsulation") encloses a first data packet in 
a new packet by appending a new header (transmitted in an unencrypted format) to 
the first data packet, so the network routes the new packet based on the information 
30 contained in the new header. The first data packet is usually encrypted when 

contained in the new data packet so no information can be gleaned from it, except 
by the intended recipient. The encapsulated packets travel through the network 

-2- 
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until they reach the destination identified in the new header. At the destination, the 
new header is stripped away and the first data packet is decrypted and processed. 
The tunneling and encryption may employ DES and 3 DES standards-based 
technology for transferring data between network locations more securely via an 
5 OC-48 TCP/IP infrastructure, for example. 

As determined by the inventors, several advantages to Internet VPNs include 
improved privacy, reduced cost relative to dedicated leased lines, and an improved 
coverage area, largely owing to the availability of the global reach of the Internet. 

10 

As recognized by the present inventors, conventional Internet VPNs are suboptimal 
in flexibility and scaleability. Figure 1, shows an example conventional VPN with 
a source probe 1 and destination probe 3 that cooperate to collect network SLA 
statistics. The source probe 1 is hosted on a personal computer using a UNIX 

1 5 operation system, for example, and has a particular IP address. The source probe 1 
prepares a 1 -packet probe (probe message) that is sent through a source router 7 
and then through the network 1 7 to the destination probe 3. The source probe 1 
includes in the probe message a time stamp, indicating the time at which the source 
probe I sent the probe message. The source router 7, which is maintained on a 

20 customers site with the source probe 1 , has a different IP address than the source 
probe 1 . The router 7 also handles signals for terminals on a source LAN 10, 
which itself has a different IP address. As with the source probe 1, source router 7 
and source LAN 10, the destination probe 3, destination router 13 and destination 
LAN 1 2 all have unique IP addresses. 

25 

The network 1 7 includes routers 9 that are interconnected by way of lines 4. 
Likewise, routers 5 are interconnected by lines 2. Interconnections between 
routers 9 and 5 are not shown to help illustrate the point that there are different 
physical paths that a packet may follow through the network 17 when traveling 
30 from the source probe 1 to the destination probe 3. The actual path that a particular 
packet follows (i.e., an "in-band" path, or channel) will be influenced by the 
source/destination pair included in its header. Beacause the source/destination pair 
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will vary depending which device is generating the packet and which device is 
receiving the packet, packets handled by the source router 7 and ultimately headed 
through destination router 13 may follow different routes through the network 17. 
Routes 5 and 9 in the network include routing tables that direct how certain packets 
5 are routed, and thus these routers may handle one packet from the source probe 1 , 
different from a packet generated by a terminal on the source LAN 10. Thus, a 
data packet from the source LAN 10 may follow a path through the routers 5 and 
lines 2 fin-band" path) while the probe message may follow a path through the 
routers 9 and lines 4 (i.e., not H in-band n ). Of course, the two paths may be the 
1 0 same, although there is no guarantee. 

The operation of sending the probe message and collecting statistics is now 
described. The probe message is formed and sent from the source probe 1 at a 
predetermined time and a time stamp of the send time is included in the probe 

1 5 message. Once the probe message is passed through the network 1 7 and by the 
destination router 1 3 to the destination probe 3, the destination probe 3 recognizes 
that the probe message has been received. The destination probe 3 then sends a 
reply probe message to the source probe 1 , and includes information in the reply 
probe message regarding the time that the destination probe 3 took between 

20 receiving the probe message and transmitting the reply probe message. Thus, the 
reply probe message includes the time stamp inserted by the source probe 1 and the 
remote latency caused by the destination probe 3. In this way, when the source 
probe 1 receives the reply probe message it is possible to determine the round trip 
time between when the source probe 1 originally sent the probe message and the 

25 time that the reply probe message was received by the source probe 1 , less the 
remote latency time. The source router 7 and the destination router 1 3 may be 
4500 CISCO routers that are configured to receive packets from both the source 
LAN 1 0 as well as the source probe 1 . Thus, the source router 7 is generic in 
operation and is a separate network component hosted in a separate housing from 

30 the source probe 1 . 

-4- 
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Availability is one of the SLA statistics that is collected by way of the probing 
process. Because availability relates to a measurement that is taken over a period 
of time (or over a number of discrete events), the source probe 1 is configured to 
set a polling interval at 2.5 minutes so as to provide two measurements for a 5 
5 minute window, and therefore provide a 5 minute resolution with regard to the 
availability statistic. 

The present inventors recognized that the VPN architecture shown in Figure 1 is 
suboptimal in that it does not offer the desired flexibility and scaleability features 

1 0 that would allow for independent upgrading and maintenance of the shared 
network 1 7. The present inventors have recognized that the shared network 1 7 
may be reconfigured and upgraded for future operations. In doing so, it is even 
possible that the additional nodes may be added to the VPN, or even the service 
level agreement may vary form time to time. Accordingly, it is a limitation with 

15 the VPN shown in Figure 1 that the source probe 1 and destination probe 3 are 
"hard-wired" to operate at certain polling intervals. Furthermore, the source and 
destination probes do not necessarily send the probe messages in-band (i.e., over 
the same physical path traversed by data packets sent between the source LAN 10 
and the destination LAN 12), even though the SLA is tied to the performance of 

20 the in-band channel. 

Accordingly, by having the source probe 1 , as well as the destination probe 2, 
implemented in a separate computer outside of the source router 7 and having a 
separate IP address, operators of the VPN are therefore limited by the capabilities 
25 of the source probe 1 to accurately collect SLA statistics. This is especially 
problematic when changes are to be made to the "core" shared network 17. 
Further more, the amount of space required to host the source probe 1, the source 
LAN 10 and the router 7, adds to maintainability restrictions at the source site. 

30 "5- 



SUBSTITUTE SHEET (RULE 26) 



WO 01/47190 



PCT/USOO/35154 



SUMMARY OF THE INVENTION 

In light of the above-discussed and other limitations of conventional systems and 
methods for collecting SLA statistics, an object of the present invention is to 
5 overcome these and other limitations by providing a software reconfigurable 
probing router. 

A feature of the present invention is to include probing router at both the source 
site and the destination site such that the probing operation is performed within the 

10 router housing itself, using processing resources available from the router. In this 
way, the probing operation is performed in software (although 
hardware/firmware/software combinations are alternatives as well) so that changes 
in the core network and SLA statistic collection processes may be quickly and 
easily accomplished. Furthermore, the probing router sends the probe message 

15 through the same path as the data, thus providing a direct measurement of SLA 
data. 

Another feature of the present invention is that an operations center connected to 
the network enables a remote "VPN builder" to remotely configure each of the 

20 probing routers in the VPN, so that within a short period of time the topology of 
the VPN may be enabled by informing each of the probing routers of the statistic 
collection obligation it has and communicating and replying to probe messages 
with other probing routers in the VPN. Furthermore, the operation center enables a 
remote probe poller processor to receive, compile, and calculate SLA statistics for 

25 the VPN. The statistics may be collected at rates consistent with the SLA for the 
particular VPN. Furthermore, the operating center enables a SLA reporting system 
to report data collected by the probe poller processor in a format that is convenient 
for the VPN customer to verify that the SLA metrics were in fact complied with 
during a particular operation cycle. 
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Other features and advantages of the present invention will become readily 
apparent form the following detailed description when read in conjunction with the 
accompanying drawings. 

5 BRIEF DESCRIPTION OF THE DRAWINGS 

A more complete appreciation of the present invention and many of the attendant 
advantages thereof will be readily obtained as the same becomes better understood 
by reference to the following detailed description when considered in connection 
1 0 with the accompanying drawings, wherein: 

Figure 1 is a block diagram of a conventional VPN that includes separate routers 
and destination probes; 

15 Figure 2 is a block diagram of a VPN that employs a probing router according to 
the present invention; 

Figures 3a-3c respectively represent data structures for a packet data unit for an 
Internet protocol packet employed as part of the present invention, as well as data 
20 structures for a probe message and reply probe message according to the present 
invention; 

Figure 4 is a block diagram of components of a probing router according to the 
present invention; 

25 

Figure 5 is a flowchart of a process for employing the probing routers so as to 
collect SLA statistics according to the present invention; and 

Figure 6 is a flowchart of a process for configuring and collecting SLA statistics 
30 information according to the present invention. 

-7- 
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DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENT 

Referring now to the drawings, specific terminology will be employed for the sake 
of clarity. However, the present invention is not intended to be limited to the 
5 specific terminology so selected and it is to be understood the each of the elements 
referred to in the specification are intended to include all technical equivalents that 
operate in a similar manner. 

Figure 2 is a block diagram of a VPN and supporting components according to the 
present invention. Data from a terminal (i.e., data source) node at a source LAN 

10 2 1 0 is sent by way of a source VPN probing router 207 through a network 2 1 7, 

which may be the Internet or another shared network, to a destination VPN probing 
router 203 (sometimes referred to as "PR") and finally to a destination LAN 208. 
The network 21 7 is a shared resource such as the Internet. However, other types of 
networks may be used that employ TCP/IP, or a related packet switched protocol 

15 such as IP version 4 or IP version 6. The physical medium in the network 217 may 
be made of any combination of terrestrial ground lines, optical lines, or wireless 
links that will form the in-band channel 204 or other channel paths 206 for 
example. Various nodes are hosted in the network 2 1 7 that may be configured to 
become part of the VPN, as will be discussed. These nodes are served by routers 

20 205 and 209 for example. For convenience, lines 204 are shown with a darker line 
indicating that this is the path through which the source LAN 210 and destination 
LAN 208 communicate with one another in a first scenario. Dynamic routing 
tables in the routers 209 and 205 dictate the path to be followed by the message 
traffic (whether encapsulated on not), where the chosen path is affected by the 

25 source/destination pair included in the message traffic header. Since the source 
VPN probing router 207 and the destination VPN probing router 203 both have IP 
addresses that may be used as header source/destination pairs in headers of 
encapsulated packets, both probe packets and encapsulated data packets will 
traverse the same path. As a consequence, the SLA statistics will be determined 

30 from in-band channel measurements since the probe message traverses the same 
path as the data packets. 

-8- 
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As can be seen, at the site where the source LAN 210 and source VPN probing 
router 207 are located, the source VPN probing router 207 need only connect to the 
source LAN 201, but not a separate source probe 1 as was the case with the 
configuration in Figure 1. The source VPN probing router 207 relays message 
5 traffic between the source LAN 2 1 0 and the network 2 1 7 according to 

conventional routing operations. In addition, the router includes program memory 
that holds therein instructions that are executed by a processor to form a probe 
mechanism that, at programmable time intervals, generates a packet data unit (a 
probe message) for transmitting through the in-band channel 204 to the destination 
1 0 router 203. The probe message includes a time stamp that indicates the time at 
which the source VPN probing router 207 actually sends the message over the in- 
band channel 204 to the destination VPN router 203. Alternatively, the time stamp 
is stored and retained by the VPN probing router 207. 

1 5 The polling interval at which the source VPN probing router 207 sends the probe 
messages is set by a VPN operation center 221 (VPNOC) that downloads a control 
instruction to the source VPN probing router 207. The control instruction includes 
an appropriate time interval indicator (polling interval) used for performing the 
probing operation. The probe message, after having passed through the in-band 

20 channel 204, is received at the destination VPN probing router 203, which 

identifies the probe message and subsequently prepares a reply probe message 
along with data (i.e., remote latency data) regarding the amount of time it took the 
destination VPN probing router 203 to prepare and send the reply probe message 
back to the source VPN probing router 207. Once the source VPN probing router 

25 207 receives the reply probe message, the source VPN probing router 207 stores 
the data contained therein but ultimately will send the data retrieved from the probe 
message in the reply probe message to the probe poller processor 223, which is 
connected to the VPNOC 22 1 . Part of the remote latency for the destination VPN 
probing router 203 is a time required to perform the tunneling operation for 

30 deencapsulating and subsequently encapsulating the probe message and reply 
probe message respectively. 

-9- 
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By implementing the probing operations in software in the source VPN router, and 
making the probing operations software reconfigurable, the system shown in 
Figure 2 is able to offer several advantages. One advantage is that separate 
hardware components are not required to perform the probing and routing 
5 operations, but rather the resources available from the router 207 are employed for 
performing the probing operation. Furthermore, the parameters of the probing 
operation are software settable, and may be remotely adjusted from the VPNOC 
221, and usually form the QVPN builder 227. One advantage of including the 
probing operations in the source VPN router is that additional components need 
1 0 not be maintained at a customer's premise. Furthermore, the processing demands 
of the probing operations are sufficiently small with respect to those performed in a 
traditional router such that sufficient processing power and memory are available 
for performing the probing operation. 

1 5 Including the probing operations within a router 207, assists in offering a more 
flexible and scalable architecture than the conventional approach. For example, 
SLA statistic collection parameters may be altered by the QVPN builder 227 by 
changing software settings in the probing router which allows for upgrades in 
equipment at the source site or the destination site to be upgraded quickly and 

20 efficiently when changes to the SLA statistic collection operation are desired. 

Furthermore, the inventive system helps to achieve a goal of isolating the functions 
performed in the network 2 1 7 from those performed at the source site or the 
destination site. Consequently, the operator of the network 217 may upgrade the 
network independent of whether any changes are made at the source site or the 

25 destination site. The isolation of the functionality performed by the network 2 1 7 
and that performed by a source equipment or destination equipment is 
accomplished by isolating "core" communication transport functions performed in 
the network 217 form node -specific operations performed at different nodes 
connected to the network, such as at the source site. In this way, the network 217 

30 may be upgraded separate from the equipment at the source site or the destination 
site. Once the core network is changed, any reprogramming of the VPN probing 
routers is accomplished by configuration commands sent from the QVPN builder. 
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The VPNOC 221 hosts the QVPN builder, which is a software-based mechanism 
used to configure VPN technology, set security profiles and distribute keys to each 
VPN site in an automatic fashion. Consequently, adding new VPN sites or adding 
5 more tunnels to the VPN is quickly performed since all of the probing routers may 
be adjusted in operation by control instructions sent from the QVPN builder. 
Accordingly, network operators do not need to manually secure IPSec tunnels for 
each of the IP nodes required to communicate over the VPN. By employing the 
VPN builder in the network architecture as shown with the use of the VPN 
1 0 probing routers 207 and 203 and other probing routers, it is possible to easily scale 
a VPN according to customer requirements. 

The probe poller processor 223, which is also hosted in the VPNCO 221, is able to 
receive SLA statistics data from the source and destination VP probing routers. 

1 5 The probe poller processor 223 then calculates an average total return time R u for 
transmission of a probe message and return of a reply probe message according to 
the equation 

Rtt=(T 2 -T,)-R L , where 
R tt is round trip time of a probe message and reply probe message, T 2 is a time at 

20 which the reply probe message is received from the destination probing router, T| 
is a time at which the probe message is sent from the source VPN probing router 
and R L is remote latency, which refers to the amount of time that the destination 
VPN probing router requires to prepare and sent the reply probe message in 
response to receiving the probe message. 

25 

The probe poller processor 223 is implemented in software and executed on a 
processor, but may also be implemented in any combination of hardware and/or 
firmware such as with an application specific integrated circuit. The probe poller 
processor 223 determines that an availability outage occurred when two adjacent 
30 packets are observed as being lost. However, other availability calculations may 
be performed as well, such as by determining availability on a packet by packet 
basis. A particular packet is viewed as being lost if R lt exceeds a predetermined 

-11- 
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amount. A packet-loss rate may be determined by observing a number of total 
packets sent within a predetermined time period, perhaps within a five minute 
window, or even a one month window, and determining a ratio of the number of 
reply probe messages received versus the number of probe messages sent. By 
5 collecting and saving packet -loss information on a packet by packet basis* 
availability may be calculated as [number of probe messages sent - dropped 
packets]/[total number of probe messages sent]. Availability may be determined in 
a variety of other ways, such as whether a predetermined number of packets are 
dropped within a predetermined period of time (for example two packets dropped 
10 in 5 minutes, where the polling interval is 2.5 minutes). 

More particular implementation details are now described. A Probe Poll List is 
maintained as an ASCII text file. This file can be called as a parameter by the 
probe poller processor on startup. If a file parameter is passed, this overrides any 

1 5 Probe Poll List maintained in a preference file. Additional probes can be 
configured directly through a configuration edit display. Through the menu 
options for this screen, the user can add, delete or import Probes to the Probe Poll 
List. The default Probe Poll List resides in the root level application directory 
called, probeList.txt. This file can be created with any standard text editor. The 

20 Probe Poll List file is organized by VPN. The VPN is defined (created) as: 
VPN = <vpn name> 

Where <vpn name> is the name of the current VPN. For each probing router 
associated with this VPN, a line of text follows to define the required probing 
25 router parameters. Each probing router parameter line begins with: 
PR = <ip address> 

Additional parameters are optional and all parameters are delimited by colons. 
Any missing parameters will be set by defaults in the application when the Probe 
30 Poll List is parsed. 

-12- 
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Parameter 


Descripti n 


NAME = <sysName> 


The sysName of the probing router. 
This is set by the probe poller 
processor during the initial poll 
sequence. 


SNMP = <version> 


V2 for snmp V2 access or V3 for 
snmp V3 access. 


COMMUNITY = <community string> 


The snmp V2 community string. 


PORT = <snmp port> 


This defaults to 161. 


TIMEOUT = <va1ue> 


The snmp timeout value for requests 
to this probing router 


RETIRES = <value> 


The snmp retry value for requests to 
this probing router. 


USER = <user name> 


The snmp V3 user id. 


AUTHPROTO = Outhentication protocol> 


The snmp V3 authentication protocal 
to use: NONE. MD5. or SHA. 


AUTHPWD = <authentication password> 


The snmp V3 authentication 
password. 



Probe Poller Processor Output Format 

The characteristics of the latency logs are as follows: 
File Name: latency.log<timestamp when file closed>.gz 

5 

Directory Structure on the Monitoring System Server: 

$VPNLOGS/vpnlogs-<collector process pid>-<sequential counter>/<probe 
hostname>/<vpn name> 

1 0 File Characteristics: ASCII, colon delimited fields, compressed with gzip, lines 
beginning with are comment fields, All timestamps are UTC, a character is 
output on the last line to terminate the file. 

-13- 
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The file contents and data structure saved in memory of each record saved in the 
VPN probing router is as follows: 



Field 


Description 


dstIP 


IP address of the remote PR to receive 
the probe packet 


dstPort 


Port on which the remote PR listens 


srcIP 


IP address of the PR probe which 
initiates the probe packet 


srcPort 


Port on which the PR probe listens 


seqstart 


Timestamp assigned when the PR probe 
initializes 


seqcount 


Next sequential counter for this remote 
PR 


send-seconds 


Timestamp when the PR probe initiates 
a probe packet 


send-ms 


Coupled with send-seconds B 
microseconds 


recv-seconds 


Timestamp when the PR probe receives 
the probe packet response from the 
remote PR 


recv-ms 


Coupled with recv-seconds- 
microseconds 


remote-process-ms 


Number of microseconds spent 
processing the sample packet on the 
remote PR to turn around a response 
packet 


flags 


Bit I indicates packet type: 
0 = data; i = test 



-14- 
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The probing routers may generate SNMP Traps when the number of packets lost in 
a predetermined amount of :ime exceeds a predetermined threshold, and if the 
probe latency is measured as exceeding a predetermined time. 

5 SLA statistical data complied by the probe poller processor 223 is provided to the 
SLA reporting system 225. The SLA reporting system 225 provides to a customer 
a condensed aggregation of data collected by the probe poller processor 223 so that 
the customer may review whether the SLA was complied with during the reporting 
interval. In one embodiment, the SLA system 225 aggregates the data on a month- 

1 0 by-month basis and provides the data via a server on an Internet web-site for 
review by customers of the VPN. Alternatively, a computer and printer are 
employed to provide written reports summarizing the SLA statistics that were 
collected for the customer of the VPN. 

1 5 The probing operations are performed on the network 2 1 7 at layer 3 (i.e., IP layer). 
Thus, the operation is performed independent of the physical and data link layers 
and thus may be used in any one of a variety of different network configurations 
such as frame relay, ATM, FDDI, packet-over SONET, Ethernet, fibre channel as 
well as others. A description of example network systems that may be employed 

20 with the current invention is provided in "Data and Computer Communications", 
by William Stallings, Fifth Ed., Prentice Hall, Chapter pages 401-458, 1997, the 
entire contents of which being incorporated herein by reference. Furthermore, 
Chapters 15 and 16 provide further description of specific protocols and 
architectures that may be employed with the present invention, and thus Chapters 

25 15-16, pages 497-584 are also incorporated herein by reference. 

While encryption may be employed to improve information privacy, encryption 
need not be employed and thus is an optional feature, selected by a customer when 
subscribing to the VPN service. The source VPN probing router 207 may also 
30 employ multi-protocol label switching that prioritizes packets through the core 
communication network 2 1 7. 
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Figure 3a illustrates a generic protocol data unit for a probe message sent by the 
source VPN probing router 207 according to the present invention. Consistent 
with the operation of TCP/IP, IP header 301a and IP data area 301b form part of an 
5 IP datagram portion of a network-level packet 303. The network-level packet 303 
includes a frame header 303a and a frame data area 303b. 



Figure 3b shows a functional description (i.e., those data fields that are relevant to 
the present probing discussion) of an IP datagram portion of the packet employed 

10 for the probe message. IP header 301a is followed by a source time stamp 32 lb, 
which is placed in the IP data area portion of the IP datagram 321 . This source 
time stamp Tl is transmitted in the probe message to the destination VPN probing 
router 203. Alternatively, the source VPN probing outer does not include the time 
stamp Tl , but does save the time stamp in memory for later use after the reply 

1 5 probe message is received. 

Figure 3c shows the IP datagram for the reply probe message. As shown, the IP 
datagram 331 includes a field 33 la that holds a measurement value (an indicator) 
of the remote latency R L as being equal to R2-R1, where R 2 is the time that the 

20 destination VPN probing router sent the reply probe message, and R| is the time at 
which the probe message was received by the destination VPN probing router 203. 
Accordingly, the remote latency R ( is the difference between these two times and 
measures the amount of time that was required by the destination VPN probing 
router 203 to generate and send the reply probe message after receiving the probe 

25 message. The reply probe message also includes the source time stamp T I 32 lb. 
The source probing router 207 then receives the reply probe message at time T2. 



Figure 4 represents the internal components of a source VPN probing router 
according to the present invention. Within a housing 401 , the probing router 
30 includes a data bus 403 that interconnects a processor 405 with other components 
connected to the bus 403. In particular, the processor 405 executes computer 
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readable instructions saved on ROM 409 to implement both a routing engine 477 
as well as the programmable probe device 407. 

The main memory 408 is a RAM that receives software settable parameters sent 
5 from the QVPN builder 227 (Fig. 2) for setting the probing parameters that would 
be executed by the programmable probe device 407. The programmable probe 
device 407 is shown to be internal to the processor 405, which is the case when it 
is implemented only in software, but may also be a separate component that 
communications with the other components by the bus, or other signal relaying 

10 mechanism, such as a local bus or optical link. The programmable prove device 
includes a timer that generates a probe message after a predetermined time has 
elapsed since the last probe message was sent. The programmable probe device 
407 either maintains internally thereto, or retrieves from main memory 408, a 
polling interval parameter that was set by the QVPN builder 227. Furthermore, the 

1 5 programmable probe device 407 also receives an indication from the QVPN 
builder 227 which destination VPN probing routers the source VPN is to 
communicate with so that tunnels may be established therebetween. 

A storage device 410 is also a RAM and is used to hold information regarding 
20 round trip delay and whether packets are dropped. This information is later sent to 
the probe poller processor 223, either on demand from the probe poller processor 
223 or at periodic intervals as a software settable parameter and saved in main 
memory 408. The packet grouping logic 41 7 and envelope packet logic 419 
cooperate to form IP packets for assessing whether received packets are to be 
25 routed to a device connected to the router, or not. Likewise, the packet grouping 
logic 417 and envelope packet logic 419 cooperate to form packets for sending 
over the IP network 4 1 7 by way of the input/output unit 4 1 5. A buffer unit 4 1 3 
serves as a buffer for saving and holding message traffic when the processor 405 is 
busy (for inbound messages) or for sending packets when either the input/output 
30 unit 4 1 5 is busy or the IP network 4 1 7 is busy. The input/output 4 1 5 connects by 
way of a bus 421 to the IP network 41 7. A local source terminal 450 also connects 

-17- 



SUBSTITUTE SHEET (RULE 26) 



WO 01/47190 



PCT/US00/35154 



to the input/output unit 415 for local accessibility to the router. The IP network 
41 7 and source terminal 450 connect through ports (or connectors) to the housing 
401. 

5 Figure 5 is a flowchart showing a process flow for collecting SLA statistics over 
the VPN. The process begins in step 501 where an inquiry is made regarding 
whether a predetermined time period has elapsed since the source VPN probing 
router has sent the last probe message. If the response to the inquiry is negative, 
the inquiry is made again until the time period has in fact elapsed. Once the 

1 0 response to the inquiry is affirmative, the process proceeds to step 503 where the 
source VPN probing router sends a polling packet to the destination VPN probing 
router 203. The polling packet (probe message) optionally includes a time stamp 
TI therein. Alternatively, the source VPN probing router simply stores in memory 
the time at which the polling packet has been sent, thus not notifying the 

15 destination VPN probing router when the message was in fact sent. 

After step 503, the process proceeds to step 505 where the probe message is 
received at a time R| at the destination VPN probing router. The destination VPN 
probing router then prepares a reply probe message and sends the reply probe 
20 message at a time R 2 such that the remote latency (i.e., turn-around time of the 
destination VPN probing router) is given by R L = &2 - R|. The process then 
proceeds to step 507 where the remote latency (or processing delay) R L is inserted 
in the reply probe message and the reply probe message is then sent. 

25 After step S507, the process proceeds to step S509 where the programmable probe 
device 407 (Figure 4) compares the amount of time between when the probe 
message was sent (Tl) and when (if at all) a reply probe message is received (T2). 
In step 509 if it is determined that the difference between T2 and Tl is greater than 
a predetermined amount (a software scttable parameter) then it is determined that 

30 the packet (probe message) was dropped. If the packet was dropped, the process 
proceeds to step S51 1 where an indication is saved in memory 410 (Figure 4), or 
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sent directly to the probe poller processor 223 (Figure 2) indicating that a packet 
was dropped. The process then proceeds to step S519. 



If however, the response to the inquiry in step S509 is negative, the process 
5 proceeds to step 5 1 3 where the time stamp T 2 is determined from when the reply 
packet (reply probe message) is received and the process then proceeds to step 
S515 where a round -trip time R« is calculated. The calculation for round-trip time 
is determined as R tt = (T 2 - TO - R L . The process then proceeds to step S5 1 7 
where R n is stored in memory at the probing router, although alternatively the data 
1 0 may be sent directly tot he probe poller processor 223 at the VPNOC 22 1 . 

The probe poller processor 223 gathers information form the respective probing 
routers in the VPN and calculates average round-trip time, R lt , availability, and 
packet loss rate for each tunnel as well as for the entire VPN. After having 

1 5 collected these SLA statistics, the process proceeds to step S52 1 where an inquiry 
is made regarding whether an SLA performance is judged to be below a required 
level, typically the service level agreement threshold levels. If the response to the 
inquiry in step 52 1 is negative, the process repeats so as to maintain a SLA 
statistical retrieval monitoring process. On the other hand, if the response to the 

20 inquiry in step 52 1 is affirmative, the process proceeds to step 523 where 

corrective action is taken on the network resources. This may include dispatching 
a trouble-shooting technician to identify a source of the problem or adjusting the 
software settable parameters in the probing router, so as to be less stringent on the 
service level requirements imposed on the network. The corrective action may 

25 also include providing a refund to a client, if the service level agreement statistics 
were in fact below the required level. After step 523 the process then repeats so as 
to continue the SLA statistic collection and analysis operation. 

Figure 6 is a flowchart of a process for automatically and remotely configuring a 
30 VPN architecture according to customer-specified requirements. The process 

begins in step S601 where the QVPN builder 227 is provided with VPN topology 
configuration information, which identifies the different VPN nodes that will be 
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used in the customer-specified VPN. The process then proceeds to step S603 
where the probing routers are either manually assigned a polling interval, or a 
default setting is included, such as two minute intervals. The process then 
proceeds to step 605 where the QVPN builder 227 sends configuration messages to 
5 the respective probing routers by way of the network 2 1 7. The probing routers 
then set the software settable parameters for the programmable probe device 407 
either in the main memory 408 or in the programmable probe device itself. 

After step S605 the process proceeds to step S607, where the programmable probe 
1 0 device 407 (Figure 4), causes the SLA statistical data that is saved in the storage 
device 41 0 to be sent to the probe poller processor 223 (Figure 2). The probe 
poller processor 223 creates a database in the probe polling processor and holds the 
data therein for calculation and distillation of SLA statistical data. 

15 In the event that changes are required in the network, the process proceeds to step 
609 where the QVPN builder 227 dispatches a "configuration" message to 
respective of the programmable probe devices in the probing routers. The 
configuration messages include the software settable parameters used by the 
probing routers to determine the polling interval, dropped packet threshold 

20 decision time, and other parameters such as particular node addresses to which to 
communicate with in determining round-trip time for packet transmission. Once 
the configuration messages are dispatched, the process proceeds to step S61 1 
where the configuration messages are received at each of the programmable probe 
devices and the programmable probe devices employ the parameters contained 

25 therein to perform probing operations at the pooling interval identified in the 
configuration message. Subsequently the configuration process ends. 

The processes and control mechanisms set forth in the present description may be 
implemented using conventional general purpose microprocessors in the routers 
30 that are programmed according to the teachings of the present specification as will 
be appreciated to those skilled in the relevant art(s). Appropriate software coding 
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can readily be prepared by skilled programmers based on the teachings of the 
present disclosure, as will also be apparent to those skilled in the relevant art(s). 

The present invention thus also includes a computer-based product that may be 
5 hosted on a storage medium and include instructions that can be used to program a 
computer to perform a process in accordance with the present invention. The 
storage medium may include, but is not limited to, any type of disk including 
floppy disks, optical disks, CD-ROMs, magneto-optical disks, ROMs, RAMs, 
EPROMs, EEPROMs, flash memory, magneto or optical cards, or any type of 
1 0 media suitable for storing electronic instructions. 

Obviously, numerous additional modifications and variations of the present 
invention are possible in light of the above teachings. It is therefore to be 
understood that within the scope of the appended claims, the invention may be 
1 5 practiced otherwise than as specifically described herein. 



20 



25 



30 
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CLAIMS; 

1 . A probing router comprising: 
a bus; 

a routing engine coupled to the bus and configured to forward packets to a 
5 communications network; 

a communication network port coupled to the bus and configured to 
connect to a communication network and transmit a probe message and the packets 
therethrough; and 

a probe mechanism configured to generate and send the probe message 
10 through said communication network port to the communication network at a time 
Tl. 



2. The probing router of Claim I, wherein: 

said probe mechanism is configured to send the probe message over an in- 
1 5 band communication channel. 

3. The probing router of Claim I, wherein: 

said probe mechanism being configured to receive a reply probe message at 
a second time, T2, sent by a destination router in response to receiving said probe 
20 message with a remote latency indicator therein so that service level agreement 
characteristics may subsequently be derived by comparing Tl , T2 and the remote 
latency indicator. 



4. The probing router of Claim 3, further comprising: 
25 a memory, wherein the probe mechanism being configured to identify and 

store in the memory the service level agreement characteristics. 



5. The probing router of Claim 2, wherein: 

said in-band channel being a tunnel channel in a virtual private network. 

30 

6. The probing router of Claim 3, wherein: 
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said reply probe message including a data field configured to hold the 
remote latency indicator that represents an amount of time between when said 
destination router received said probe message and when said destination router 
sent said reply probe message. 

7. The probing router of Claim I , wherein: 

a polling interval at which said probe mechanism sends said probe message 
being a remotely programmable setting. 

8. The probing router of Claim 4, wherein: 

said probe mechanism being configured to send at least on of Tl , T2 t and 
the remote latency indicator to a probe poller device that calculates service level 
agreement statistics. 

9. The probing router of Claim 8, wherein: 

said probe mechanism being configured to calculate a round trip time from 
Tl, T2, and the remote latency indicator for a predetermined number of probing 
cycles and being configured to send the round trip time to a probe poller device 
that calculates service level agreement statistics. 

1 0. The probing router of Claim 9, wherein: 

said probe mechanism being configured to save an indication that a packet 
is lost for a probing cycle when said probe mechanism determines that the round 
trip time exceeds a predetermined threshold. 

1 1 . The probing router of Claim 8, wherein: 

said probe mechanism being configured to calculate service level 
agreement statistics based on Tl , T2, and remote latency, said service level 
agreement statistics including at least one of a network availability statistic and a 
packet loss rate. 
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1 2. A computer-readable medium carrying one or more sequences of one 
or more instructions for sending a probe message, the one or more sequences of 
one or more instructions including instructions which, when executed by one or 
more processors, cause the one or more processors to perform the steps of: 

5 (a) preparing a probe message; and 

(b) sending said probe message over an in-band communication channel. 

1 3. The computer-readable medium according to Claim 12, wherein the 
probe message includes a time stamp, Tl , representing when said probe message is 

1 0 sent in said sending step. 

14. The computer-readable medium according to Claim 13, wherein when 
the one or more instructions are executed by the one or more processors cause the 
one or more processors to further perform the steps of: 

1 5 receiving at a second time, T2, a reply probe message sent from a 

destination probing router; and 

extracting a remote latency indicator from said reply probe message, said 
remote latency indicator representing an amount of time between when said 
destination probing router received said probe message and when said destination 

20 probing router sent said reply probe message. 

1 5. The computer-readalbe medium of Claim 14, wherein when the one or 
more instructions are executed by the one or more processors cause the one or 
more processors to further perform the step of: 

25 calculating service level agreement statistics associated with the in-band 

communication channel of the virtual private communication network for Tl, T2 
and said remote latency indicator. 

16. The computer-readable medium of Claim 12, wherein said in-band 
30 channel being an in-band channel of a virtual private network. 

1 7. A communication system for gathering traffic statistics, comprising: 
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a probing router configured to prepare performance statistics information; 

a probe poller processor configured to receive performance statistics 
information collected by a probing router that sends a probe message through an 
in-band channel; and 
5 a reporting mechanism coupled to said probe poller processor and 

configured to present a compilation of said performance statistics information for 
comparison against performance thresholds of a service level agreement. 



18. The system of Claim 17, wherein said in-brand channel being in a 
1 0 virtual private network. 

19. The system of Claim 1 7, wherein said probing router being within a 
customer premise. 

15 20. The system of Claim 1 7, wherein said reporting mechanism being 

configured to report said performance statistics information in at least one of a 
printed form and a graphically displayed form. 

21 . The system of Claim 1 7, wherein said reporting mechanism being 
20 configured to report said performance statistics on an Internet web site. 

22. The system of Claim 1 7, further comprising: 

a virtual private network builder configured to receive topology 
information regarding an assignment of probing routers to the virtual private 
25 network and produce a control signal to be distributed to respective probing 
routers, said probing router being one of said probing routers. 



23. The virtual private network operation center of Claim 22, wherein: 
said control signal including a polling interval indicator that sets a polling interval 
30 at which said probing router sends said probe message. 



24. The virtual private network operation center of Claim 1 7, wherein: 
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said probe poller processor being configured to calculate at least one of an 
availability and a packet loss rate of said in-band communication channel from 
said performance statistics information. 

5 25. A probing router comprising: 

means for routing data packets within a virtual private network; 
means for preparing and sending a probe message through an in-band 
channel of the virtual private network; and 

an enclosure that houses said means for routing and said means for 
1 0 preparing and sending. 

26. A method for collecting network performance statistics, comprising the 
steps of: 

(a) preparing a probe message with a probing router; 
15 (b) sending said probe message over an in-band communication channel; 

and 

(c) measuring a propagation time for said message to reach a predetermined 
location. 

20 27. The method of Claim 26, wherein said sending step comprises sending 

said probe message in the in-band communications channel of virtual private 
network. 

28. The method of Claim 26, wherein: 
25 said preparing step comprises including in said probe message a time 

stamp, Tl, indicative of when said probe message is sent in said sending step; and 
said measuring step includes 

receiving a reply probe message sent from a destination probing 
router at a second time, T2, said destination probing router being located at said 
30 predetermined location, and 

extracting a remote latency indicator from said reply probe 
message, said remote latency indicator representing an amount of time between 
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when said destination probing router received said probe message and when said 
destination probing router sent said reply probe message. 

29. The method of Claim 26, further comprising a step of: 

5 calculating a service level agreement statistic from said propagation time. 

30. The method of Claim 26, further comprising the steps of: 
sending said service level agreement statistic from a probing router to a 

network operation center; 
1 0 combining at said network operation center said service level agreement 

statistic with other service level agreement statistics from other in-band channels to 
provide a compilation of service level agreement statistics; and 

presenting said compilation of service level agreement statistics to a user of 
said virtual private network. 

15 
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25 



30 
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